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Abstract 

■P 

This  report  explains  the  term^noise'"  from  the  point 
of  view  of  probability  and  statistics.  Namely,  noise  is 
regarded  as  a  random  process.  Two  special  cases  of  noise, 
Gaussian  noise  and  white  noise,  are  discussed. 
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The  term  "noise"  was  first  used  in  communications 
engineering  as  a  result  of  the  undesired  acoustic  effects 
accompanying  spontaneous  electric  fluctuations  in  receivers. 
Ever  since  the  advent  of  electric  communications,  communi¬ 
cations  engineers  have  searched  to  reduce  the  electric  noise 
in  communications  systems.  Nowadays  the  idea  of  noise  has 
been  widely  used  in  many  other  fields  where  no  acoustic 
effect  is  involved.  Roughly  speaking,  noise  means  something 
which  interferes  with  the  desired  signal.  There  is  a  variety 
of  sources  of  noise.  For  example,  the  fundamental  source  of 
electric  noise  is  the  quantized  electric  charge.  (Its 
graininess  causes  the  voltage  fluctuations  in  electric 
circuits.)  Another  common  example  is  the  computer  generated 
noise  due  to  roundoff  error.  In  many  statistical  methods  such 
as  regression  analysis  end  categorical  data  analysis,  the  noise 
is  the  randomly  fluctuating  part  (e.g.  due  to  sampling)  and  the 
signal  is  the  unknown  deterministic  parameters.  To  understand 
the  noise  structure  and  to  extract  the  signal  from  the  noisy 
environment  are  the  most  important  Objectives  in  many  studies. 

Noise  is  usually  treated  as  a  stochastic  process  of 
irregular  fluctuations.  In  applications,  the  noise  process 
is  often  assumed  to  be  stationary  and  ergodic*.  That  is, 
its  statistical  properties  can  be  completely  characterized 
by  just  one  sample  over  a  long  period.  Many  tools  used  in 
tins  series  analysis*  [4,  Fart  II]  are  very  useful  in 
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analyzlng  noise.  For  example,  fast  Fourier  transform*  (FFT) 

[10,  Chapter  6]  is  a  powerful  computational  tool  in  estimating 

noise  power  spectra.  The  analysis  of  autocorrelation  and 

* 

partial  autocorrelation  functions  is  useful  in  fitting 
auto regressive-moving  average*  (ARMA)  models  to  the  noise 
process. 

GAUSSIAN  NOISE 

In  many  cases,  noise  is  best  described  as  a  Gaussian 
process*  (e.g.  thermal  noise  and  shot  noise  in  electronic 
systems).  That  is,  the  joint  distribution  of  noise  random 
variables  at  any  set  of  time  points  is  multivariate  normal. 

In  other  words,  the  noise  process  can  be  completely  charac¬ 
terized  by  its  autocorrelation  function,  or  equivalently  its 
spectral  distribution  if  it  is  stationary.  The  assumed 

* 

normality  can  be  justified  by  the  central  limit  theorem  if 
the  noise  is  composed  of  many  small  independent  (or  weakly 
dependent)  random  effects.  The  main  advantage  of  using 
Gaussian  assumption  is  that  the  best  linear  estimator  of  the 
signal  is  optimal  under  the  criterion  of  mean  squared  error. 
(Of  course,  the  signal  needs  to  be  Gaussian,  too).  That  is 
to  say,  there  is  no  need  to  consider  non-linear  theory  in 
signal  estimation  in  Gaussian  systems.  In  signal  detection, 
the  likelihood  ratio  statistic  is  an  optimal  test  statistic, 
and  many  results  have  been  established  on  absolute  continuity 
and  the  Radon- Nikodym  derivative  between  the  two  measures 
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induced  by  pure  noiae  and  signal  plus  noise,  when  both 
measures  are  Gaussian  [7,  Chapter  3jl3],  For  example,  Feldman 
and  Hajek  derived  the  dichotomy  theorem  that  two  Gaussian 
measures  are  either  singular  or  equivalent.  Rice  [11,12] 
studied  the  behavior  of  sample  paths  of  stationary  Gaussian 
noise  with  zero  mean  in  continuous  time.  Suppose  that  X(t) 
is  stationary  Gaussian  with  EX(t)  ■  0  and  EX(t)X(t+s)  *  R(s) . 
Die  expected  number  of  zeros  per  second  in  sample  paths  is 

00  00  m 

2[/f2w(f)df/  / w(f)df]1/2  where  w(f)  -  4  /R(t)cos  2irftdt 

is  the  power  spectral  density  and  f  is  frequency  in  cycles 

per  second.  However,  there  is  not  much  known  about  the 

distribution  of  the  distance  between  two  successive  zeros. 

Die  expected  number  of  local  maxima  per  second  in  sample  paths 
00  00 

is  [/  f4w(f)df //  f2w(f)df]1/2.  When  the  noise  X(t)  is 
o  o 

narrow-band  (i.e.  the  spectrum  w(f)  vanishes  except  in  a 
small  region) ,  the  envelope  of  X ( t)  has  the  Ray leigh  distri¬ 
bution  which  is  defined  to  be  the  distribution  of  the  square 
root  of  a  random  variable  with  the  chi-squared  distribution 
of  degree  2. 

It  is  worth  noting  that  even  though  optimal  statistics 
can  be  obtained  under  the  Gaussian  condition,  robust 
statistics*  are  desired  so  that  decisions  based  on  the 
statistics  are  less  sensitive  to  the  Gaussian  assumption. 
Various  kinds  of  noci-Gaussian  noise  take  place  in  different 
situations.  8 os*  are  generated  from  Gaussian  noise  through 
non-linear  devices.  For  example,  the  output  voltage  of 


noise  has  the  Rayleigh  distribution  when  narrow-band  Gaussian 
thermal  noise  is  applied  to  an  evelope  detector.  Another 
example  is  quantisation  noise  which  has  the  uniform  distri¬ 
bution  from  -d/2  to  d/2  where  d  is  a  unit  quantization 
step.  Quantisation  noise  occurs  when  analog  signals  are 
converted  to  digital  form.  One  more  example  is  impulse  noise. 
Impulse  noise  is  a  generalised  random  process  conposed  of 
short  bursts  which  occur  at  random  time  points  with  random 
amplitudes.  In  the  analysis  of  electroencephalographic  (EEG) 
wave  recordings  of  brain  activity,  the  EEG  recordings  are  a 
Gaussian  signal  process  of  brain  activity  plus  a  Poisson 
process  of  impulse  noise  due  to  muscle  contractions  by  nerve 
impulses  [8]. 

WHITE  NOISE 

White  noise  is  a  stationary  stochastic  process  with 
constant  spectral  density.  The  term  "white”  is  borrowed 
from  optics  where  "white  light”  has  been  used  to  signify 
uniform  energy  distribution  among  the  colors.  (Actually 
the  analogy  is  not  correct  since  in  optics  the  uniform 
energy  distribution  of  white  li$it  is  based  on  wavelength 
(the  reciprocal  of  frequency)  rather  than  frequency 
(3,  p.  141.)  Discrete-time  white  noise  is  simply  an  uncor¬ 
related  wide-senee  stationary  time  series  with  sero  mean. 
Autoregressive- moving  average*  (AMA)  processes  are  driven 
by  discrete- time  white  noise.  Continuous- time  white  noise 
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X(t)  satisfies  formally  EX{t)  »  0  and  EX(t)X(s)  ■  a26 (t-s) 
where  £(•)  is  the  Dirac  delta  function*.  In  the  following/ 
we  only  consider  continuous- time  white  noise.  Since  entirely 
flat  spectral  density  distribution  implies  infinite  power# 
white  noise  does  not  exist  in  practice.  Nevertheless#  the 
use  of  a  white  noise  model  is  justified  in  many  aspects. 

Many  real  data  sets#  such  as  aircraft  flight  test  data# radar  re¬ 
turn  data  and  passive  sonar  detection  data#  involve  an  additive 
random  noise  which  has  large  bandwidth  compared  to  that  of 
the  signal.  In  some  other  problems#  noise  may  be  best 
described  as  a  linear  transformation  of  white  noise.  More 
importantly,  it  is  often  much  easier  analytically  and 

* 

computationally  to  deal  with  white  noise  (e.g.  Kalman  filter  ). 

A  crucial  problem  of  robustness  arises:  Can  the  results  based 
on  a  white  noise  model  effectively  approximate  those  based  on 
a  real  case  where  the  noise  involved  has  large  but  finite 
bandwidth?  When  only  linear  operations  on  the  data  are 
considered,  the  answer  is  yes#  i.e.  the  results  based  on  a 
white  noise  model  are  consistent  with  the  asymptotic  case 
where  the  noise  bandwidth  tends  to  infinity  in  any  way 
desired.  However#  serious  difficulties  arise  in  interpre¬ 
tation  when  non-linear  operations  on  the  data  have  to  be 
considered  [2].  the  problem  of  robustness  has  yet  to  be 
further  investigated  in  the  non- linear  case. 

Let  us  take  a  close  look  at  the  following  non-linear 
filtering  problem.  Let  Z(t)  and  X(t)  (t  >,  o)  be  the  mutually 
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in dependent  signal  and  white  noise,  respectively.  Let  Y(t) 

=  Z(t)  +  X(t)  be  the  observation  process.  The  classic 
filtering  problem  is  to  calculate  the  conditional  expectation 
Cor  distribution)  of  Z(t)  given  {Y(s):  o  _<  s  _<  t}.  Tradi¬ 
tionally,  white  noise  X(t)  is  treated  as  the  formal  derivative 

of  the  Wiener  process  W(t)  »  f  X(s)ds.  The  Wiener  process 

o 

is  well  defined  in  terms  of  the  Wiener  measure  on  C[0,T] 

(*  the  set  of  all  continuous  real-valued  functions  on  [0,T]) 
where  T  is  the  time  span.  Now,  the  filtering  problem  can  be 
solved  using  Ito  integral  which  allows  non-linear  operations. 
The  conditional  expectation  of  Z(t)  given  {Y(s):  o  _<  s  _<  t) 
is  uniquely  determined  up  to  a  null  subset  of  sample  paths, 
unfortunately,  the  Wiener  process  sample  paths  are  of  bounded 
variation  with  probability  zero  and  all  the  physical  sample 
paths  are  of  bounded  variation.  Thus  non-linear  filtering 
theory  cannot  be  applied  in  practice  unless  we  are  able  to 
choose  a  particular  version  of  the  conditional  expectation 
which  is  defined  everywhere  (not  just  almost  everywhere)  and 
continuous  in  sample  path  with  respect  to  the  sup remum  norm 
on  C[0,T] .  This  is  another  robustness  problem,  it  has  been 
taken  up  by  several  authors  [5,6,9].  Balakrishnan  [2]  took 
a  different  functional  approach  which  avoids  the  above 
problem.  He  developed  non-linear  white  noise  theory  in 
which  white  noise  is  defined  on  L2[0,T]  («  the  set  of  all 
square  integrable  real-valued  functions  on  [0,T] ) .  Specifi¬ 
cally,  let  C  be  the  class  of  cylinder  sets  in  L2(0,T]  with 
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Borel  bases  in  finite  dimensional  subspaces.  A  weak 

distribution  is  a  finitely  additive  probability  measure  on 

C  which  is  countably  additive  on  any  class  of  cylinder  sets 

with  bases  in  the  same  finite  dimensional  subspace.  Now, 

white  noise  is  defined  to  be  the  triple  (L2 [0,T] ,C,uG)  where 

UG  is  the  weak  distribution  (Gaussian  measure)  defined  by 

* 

the  characteristic  function 

T  T 

fL  [0,T]expli  J  h(t)x(t)dt]dMG(x)  »  exp[  -  ^  Jh2(t)dt] 
2  o  o 

for  all  heL2[0,T].  Based  on  this  construction  of  white  noise, 
some  results  on  likelihood  ratio,  innovation  process  and 
conditional  density  are  derived  (1,  Chapter  6;  2]. 

SIGNAL  PROCESSING  IN  THE  PRESENCE  OF  NOISE 

The  effect  of  noise  on  information  transmission  is  the 

* 

main  subject  of  statistical  communication  theory  .  Infor- 
* 

mation  theory  ,  signal  detection  and  signal  estimation 

(filtering  and  smoothing)  are  three  important  topics.  See 

the  article  on  STATISTICAL  COMMUNICATION  THEORY  for  a  detailed 

* 

exposition.  Spline  approximation  is  another  subject  related 
to  signal  estimation  in  the  presence  of  noise.  It  is  a 
method  of  recovering  a  smooth  (signal)  function  f(t) 

(o  _<  t  _<  T)  when  only  discrete,  noisy  measurements 

yA  “  f(t^)  +  c(tA)  (i  ■  l,2,...,n)  are  available.  What  is 

known  about  f(t)  is  that  it  is  smooth,  e.g.  f" (t)  e  L2(0,T]. 
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The  spline  estimator  of  f  is  the  spline  function  which 
1  n  2  ft  2 

minimizes  ±  I  (f(t^)  -  y^)  +  Xl  (fH(s))  ds  where  the  smoothing 

parameter  X  is  determined  by  the  generalized  cross-valida¬ 
tion  method  [14].  This  estimator  may  be  regarded  as  a  low- 
pass  smoother  in  a  general  sense.  The  estimator  has  the 
remarkable  property  that  the  higher  order  derivatives  of  it 
are  good  estimators  of  those  of  f. 


REFERENCES 


[1]  Balakrishnan ,  A.V.  (1976).  Applied  Functional  Analyeie , 
Springer-Verlag,  New  York. 

[2]  Balakrishnan,  A.V.  (1980).  Multivariate  Analyeie-V, 

P.R.  Krishnaiah,  ed. ,  North-Holland,  Amsterdam, 

pp.  97-109. 

[3]  Bennett,  W.R.  (I960).  Electrical  Noise,  McGraw-Hill, 
New  York. 

[4]  Box,  G.E.P.  and  Jenkins,  G.M.  (1976).  Time  Series 
Analyeie :  Forecasting  and  Control ,  Holden-Day,  San 
Francisco. 

[5]  Clark,  J.M.C.  (1978).  Communication  Syeteme  and  Random 
Proaeee  Theory,  J.K.  Skwirzynski,  ed. ,  NATO  Advances 
Study  Institute  Series,  Sijthoff  and  Noordhoff,  Alphen 
a an  den  Rijn,  pp.  721r734. 

[6]  Davis,  M.H.A.  (1982).  Theory  of  Proba.  and  its  Appl 
27,  pp.  167-175. 

[7]  Ibragimov,  I. A.  and  Rozanov,  Y.A.  (1978).  Gaueeian 
Random  Prooeeeee,  Springer-Verlag,  New  York. 

[8]  Johnson,  T.L. ,  Feldman,  R.G.  and  Sax,  D.S.  (1973). 
American  J.  EEC  Tech.,  13,  13-35. 

[9]  Kushner,  H.J.  (1979).  Stoohaetioe,  3,  pp.  75-83. 


9 


[10]  Oppenheixn,  A.V.  and  Schafer,  R.W.  (1975).  Digital 
Signal  Prooaeeing,  Prentice-Hall,  Englewood  Cliffs, 

Hew  Jersey. 


[11] 

Rice,  S.O. 
283-332. 

(1944). 

Ball  Syetam 

Taoh.  J. , 

23, 

pp. 

[12] 

Rice,  S.O. 
156. 

(1945). 

Ball  Syetam 

Taoh.  J., 

24, 

pp. 

46- 

[13] 

Shepp,  L.A. 

(1966). 

Ann.  Math. 

Statist. , 

37, 

pp. 

32: 

354. 

[14]  Wahba,  6.  (1977).  SIAM  J.  Bum**.  Anal.,  14,  pp.  651 
667. 


(GAUSSIAN  PROCESS 
NON-LINEAR  FILTERING 
SPLINE  APPROXIMATION 
STATISTICAL  COMMUNICATION  THEORY 
TIME  SERIES  ANALYSIS 
WIENER  PROCESS) 


read  mtntuenoN* 

BEFORE  COMPHTWO  F OHM 


security  clabbi riCATiON  or  this  raoe  rm>m  dm*  Smno 

1  REPORT  DOCUMENTATION  PAGE 

REAS  OfiTRUCTiONS  J 

BEFORE  CORPLETWQ  FORM  1 

4.  title  <m*  SuMItlt) 

Noise 

S.  TYRE  or  RBRORT  *  RERIOO  COVERED 

Technical  Report 

S.  RERroRMINO  ORO.  RERORT  NUMBER 

T  author?*)  “““ 

Yi-Ching  Yao 

S.  CONTRACT  OR  DRANT  NUMBER?,) 

N000 14-75- C-0555 

I-  Ierrorming  orbani Nation  name  and  adoresI 

Massachusetts  Institute  of  Technology 

Statistics  Center 

CaahrldgB.  ma  02139 _ _ 

Office  of  Naval  Research 

Statistics  and  Probability  Code  436 

ArUnoCnn.  VA  22217 

11.  RBRORT  DATE 

Seotember  1983 

is.  number  or  raoes 

10 

MONITORING  AGENCY  NAME  S  AOORESSflf  61  If, Wit  In  CmUMItm*  Off!,,) 

IS.  SECURITY  CLASS.  f,f  (hi,  NRMIJ 

|  OtSTmm/TlOM  STATtrntMT  rot  IM$  M wppft)  1 

This  document  has  been  approved  for  puhlic  release  an&  sale; 
its  distribution  is  unlimited. 


<7.  distribution  statement  oi  /»•  atomt  mihn  in  at,,*  so,  n  <au*rmn  tnm  r<ro «) 


IB.  SURRLEMENTARY  NOTES 


l 

i 


It’  KEY  VOffOf  m  rtm**  H  ME  ttmrtty  If  Mn*  fwilf) 

non-linear  filtering,  Gaussian  noise,  power  spectral  density,  white  noise 


20.  *  ACT  fCwHn  «  rawN  It  MCHtarr  «Mf  IMOIf  hr  M«<*  wlw) 

This  report  explains  the  term  "noise"  from  the  point  of  view  of  probability 
and  statistics.  Namely,  noise  is  rsgarded  as  a  random  process.  Two 
special  cases  of  noise,  Gaussian  noise  and  white  noise,  are  discussed. 


00  ,'STt.  1473  edition  or  *  nov  u  ii  obsolete 

5  N  O’O? •  -*■  )»<•  460  1 


Unclassified 

becu'rity  CLAturiCATiON  or  tnh  rage  .gm*  BnTKmO 


